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Abstract. The spectral properties of a class of non-selfadjoint second order 
elliptic operators with indefinite weight functions on unbounded domains Q 
are investigated. It is shown that under an abstract regularity assumption 
the nonreal spectrum of the associated elliptic operator in L 2 (Q) is bounded. 
In the special case that Q = R n decomposes into subdomains and f!_ 
with smooth compact boundaries and the weight function is positive on f2_|_ 
and negative on £7_, it turns out that the nonreal spectrum consists only of 
normal eigenvalues which can be characterized with a Dirichlet-to-Neumann 
map. 



The present paper is concerned with the spectral properties of partial differential 
operators associated to second order elliptic differential expressions of the form 



with variable coefficients ajk, a, and a weight function r defined on some bounded 
or unbounded domain ft C M" , n > 1 . It is assumed that the differential expression 
£ is formally symmetric and uniformly elliptic. The peculiarity in this paper is that 
the function r is allowed to have different signs on subsets of positive Lebesgue 
measure of fl. For this reason C is said to be an indefinite elliptic differential 
expression. 

The differential expression I in (jl.ip gives rise to a selfadjoint unbounded oper- 
ator A in the Hilbert space L 2 (tt) which is defined on the dense linear subspace 
domA = {/ £ : i(f) £ L 2 (Q)}. The spectral properties of the elliptic differ- 

ential operator A depend on the geometry of and the coefficients ajk and a, and 
are, at least from a qualitative point of view, well understood: The selfadjointness 
and ellipticity of A imply that the spectrum of a(A) is contained in R and that it 
is semibounded from below. If the domain f2 is bounded or "thin" at oo, then the 
resolvent of A is compact and hence cr{A) consists of a sequence of eigenvalues with 
finite dimensional eigenspaces which accumulates to +00; see, e.g., [15]. For general 
unbounded domains a {A) may also contain continuous and essential spectrum of 
rather arbitrary form. However, if, e.g., the coefficients ajk and a converge to a 
limit for \x\ — > 00, then the essential spectrum of A consists of a single unbounded 
interval. 

In contrast to the selfadjoint case the spectral properties of the non-selfadjoint 
indefinite elliptic operator 



associated to the differential expression in (|1.2p are much less understood, in partic- 
ular, if the domain fi is unbounded. The case of a bounded domain f2 is discussed 
in, e.g., [211 122) . where the point of view is similar to ours. Further properties of 
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indefinite elliptic operators on bounded domains, as, e.g., asymptotical behaviour 
of eigenvalues or Riesz basis properties of eigenfunctions have been studied (also 
for more general elliptic problems involving indefinite weights) in various papers. 
We mention here in particular the works [20l |23l [24l H3 ISH [27] of M. Faierman and 
EM Hi EH El] of S.G. Pyatkov, and, e.g., [3DHI2S]. 

The main objective of the present paper is to study spectral properties of non- 
selfadjoint indefinite elliptic operators of the form (|1.2j) on unbounded domains. 
Such problems are more difficult to investigate and a purely abstract operator 
theoretic and functional analytic approach is insufficient in this situation (since, 
e.g., the essential spectrum of A is in general nonempty it is difficult to conclude 
that the spectrum of T does not cover the whole complex plane). Therefore, in this 
paper we combine methods from the classical theory of elliptic differential equations 
with modern spectral and perturbation techniques for unbounded operators which 
are symmetric with respect to an indefinite inner product. Our investigations lead 
to new insights and results on the spectral properties of indefinite elliptic operators 
on unbounded domains, e.g., we prove that under an abstract regularity assumption 
the nonreal spectrum of T is bounded. Furthermore, in the special case where 

= R™ decomposes into subdomains 0+ and f2_ with smooth compact boundaries 
such that the weight function r is positive (negative) on Q + (f2_, respectively) it is 
shown that the nonreal spectrum of T consists only of normal eigenvalues which can 
be characterized with Dirichlet-to-Neumann maps acting on interior and exterior 
domains. 

The paper is organized as follows. After the precise assumptions and basic facts 
explained in Section E2 the known case of a bounded domain fl is discussed in 
Section [3] for completeness, see, e.g., [16j [22J [23j [39] . As one might expect it turns 
out that in this case the resolvent of T is compact and hence cr(T) consists only of 
eigenvalues with finite multiplicity. Some additional facts on selfadjoint operators 
with finitely many negative squares in indefinite inner product spaces from [151 
1341 135] imply that the nonreal spectrum of T consists of at most finitely many 
eigenvalues. Section [4] deals with general unbounded domains. If the spectrum or 
essential spectrum of A is positive, then again abstract methods ensure that the 
nonreal spectrum of T is bounded and consists of at most finitely many eigenvalues; 
cf. QUI E3 DS EH [Ml HO and Theorems EU and [O] One of our main results in 
the present paper states that without further assumptions on the operator A the 
nonreal spectrum of T remains bounded if a certain isomorphism W which ensures 
the regularity of the critical point oo exists; cf. condition (I) in Theorem 14.41 In 
Section 03 the special case ft = R™ with r having negative sign outside a bounded 
set is studied. A sufficient condition in terms of the weight function r is given such 
that the nonreal spectrum of T is bounded. A more detailed analysis is provided in 
Theorem l5.4l where a multidimensional variant of Glazmans decomposition method 
is used to show that the nonreal spectrum of T consist only of eigenvalues with finite 
multiplicity which may accumulate to certain subsets of the real line. Finally, it is 
shown in Theorem 15.61 how the nonreal spectrum of T can be characterized with 
the help of Dirichlct-to-Neumann maps acting on interior and exterior domains and 
a variant of Krein's resolvent formula for indefinite elliptic differential operators is 
obtained in Theorem 15.71 

2. Elliptic differential operators in L 2 (£l) 

In this preliminary section we define an elliptic differential expression C with 
an indefinite weight function on some domain 17 and we associate an unbounded 
differential operator in L 2 (fl) to L which is selfadjoint with respect to an indefinite 
metric on L 2 (f2); cf. Theorem 12. 11 
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2.1. The elliptic differential expression. Let 51 C M™ be a domain and let £ be 
the "formally selfadjoint" uniformly elliptic second order differential expression 

(2.1) (£f)(x):=- J2 (*) + 

with bounded coefficients a^. € C°°(ft) satisfying ajk(x) = akj(x) for all x £ £7 and 
j,k = 1, . . . , n, the function a € L°°(f2) is real valued and 

n n 

j,fc=l fc=l 

holds for some C > 0, all £ = (fi, . . . , £„) T G R n and a: G fi. 

In the following we investigate operators induced by the second order elliptic 
differential expression C with the indefinite weight r defined by 

(Cf)(x):=^-(£f)(x), ie!l. 
r(x) 

Throughout this paper it is assumed that r is a real valued function such that 
r, r^ 1 G L°°(f2) and each of the sets 

(2.2) fl + := {a; G tt : r{x) > 0} and := {x G H : r(a;) < 0} 

has positive Lebesgue measure. Observe that f2\(f2 + U 0_) is a Lebesgue null set. 
The restriction of the weight function r onto Q± is denoted by r±. Similarly, for 
a function / defined on il the restriction onto f2± is denoted by /±. Moreover, £± 
and C± stand for the restrictions of the differential expressions £ and C onto VL± . 

2.2. Differential operators in L 2 (fl) associated to £ and C. To the differential 
expression £ we associate the elliptic differential operator 

(2.3) Af:=£(f), domA={f€Hi(tl):£(f)€L 2 (n)}, 

where Hq(£1) stands for the closure of C^°(fl) in the Sobolev space _ff 1 (f2). It 
is well known that A is an unbounded selfadjoint operator in the Hilbert space 
(L 2 (il), (•, •)) with spectrum semibounded from below by essinfa. This can be 
seen, e.g., with the help of the sesquilinear form associated to £ and the first repre- 
sentation theorem from }31j . 

Besides the Hilbert space inner product (•, •) in L 2 (Q) we will make use of the 
indefinite inner product 

(2.4) [f,g] := f f(x)gjxjr(x)dx, f,geL 2 (Cl). 

The space (L 2 (f2), [•, •]) is a so-called Krein space; cf. [51 H31 1331 E31 133 - Observe that 
[•, •] is nonpositive on functions with support in f2_ and nonnegative on functions 
with support in + . Note also that the assumptions r G L°°(Q,) and r" 1 G L°°(0) 
imply that the multiplication operator Rf = rf, f G L 2 (f2), is an isomorphism in 
L 2 (n) with inverse R~\f = r -1 /> / e L 2 {Vl). In particular, £{f) G L 2 (n) if and 
only if £(/) € L 2 (f2). Furthermore, the inner products (•, •) and [•, •] are connected 
via 

(2.5) [f,9] = (Rf,9) and (/,<?) = [iT 1 /,.?] for f,geL 2 (n). 

Next we introduce the differential operator T associated to the indefinite elliptic 
expression £ and we summarize some of its properties. The following theorem is a 
direct consequence of (|2.5[) and the selfadjointness of A. 
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Theorem 2.1. The differential operator 

(2.6) Tf :=£(/), domr^/e^^i^/jeL 2 ^)}, 

is selfadjoint with respect to the Krein space inner product [•, •] m L 2 (r2), and T is 
connected with the elliptic differential operator A in (|2.3[) iraa 

T = i?" 1 /! and A = RT. 

We remark that the adjoint of an (unbounded) operator with respect to a Krein 
space inner product is defined in the same way as with respect to a usual scalar 
product. Here the adjoint T + of T with respect to [•, •] can equivalently be defined 
by T+ := R^ 1 A* = where * denotes the adjoint with respect to (•, ■). In 

particular, this implies [Tf,g] — [f,Tg] for all f,g£ domT. 

We also point out that the spectrum of an operator which is selfadjoint in the 
Krein space (L 2 (f2), [•, •]) can be quite arbitrary. In particular, the spectrum is in 
general not a subset of R and simple examples show that the spectrum can be empty 
or cover the whole complex plane. However, the nonreal spectrum is necessarily 
symmetric with respect to the real line. 

2.3. Spectral points of closed operators. Let S be a closed operator in a 
Hilbert space. The resolvent set p(S) of S consists of all A G C such that S — A 
is bijective. The complement of p(S) in C is the spectrum o~(S) of S. The point 
spectrum o~ p (S) is the set of eigenvalues of S 1 , i.e., those A £ C for which S — A is 
not injective. An eigenvalue A is said to be normal if A is an isolated point of o~(S) 
and its (algebraic) multiplicity is finite. The essential spectrum o~ css (S) consists of 
those points A e C for which S — A is not a Semi-Fredholm operator. Recall that 
the essential spectrum is stable under compact and relative compact perturbations; 
cf. [19j|3T]. If S is a selfadjoint operator, then a ess (S) consists of the accumulation 
points of o~(S) and the isolated eigenvalues of infinite multiplicity; the set of normal 
eigenvalues is the complement of a ess (S) in a(S). Recall that the eigenvalues of a 
selfadjoint operator are semisimple. We say that the positive (negative) spectrum of 
(a not necessarily selfadjoint operator) S has infinite multiplicity if <r(S) (1 (0, +oo) 
(o~(S) H (— oo, 0), respectively) contains infinitely many eigenvalues or points of the 
essential spectrum of S. 

3. Spectral properties of indefinite elliptic operators on bounded 

DOMAINS 

In this section we study the spectral properties of the indefinite elliptic operator 
T in Theorem 12. II in the case that O is a bounded domain in R™. Throughout this 
section it will be tacitly assumed that O is bounded, but no further (regularity) 
assumptions on the boundary are imposed. 

Let us first recall the following well-known theorem on the qualitative spectral 
properties of the selfadjoint elliptic operator A which is essentially a consequence 
of the compactness of the embedding of Hq(Q) into L 2 (£l) (see, e.g., gH Theo- 
rem 7.1]), the ellipticity of £ and the boundedness of the coefficient a. 

Theorem 3.1. The spectrum of A is bounded from below and consists of normal 
semisimple eigenvalues which accumulate to +oo. 

The principal result in this section is the following theorem, which is well known 
and follows from the more general and abstract considerations in [531 132] and 
p~5l [34l [35] . For the convenience of the reader a short proof is included. 

Theorem 3.2. The spectrum ofT consists of normal eigenvalues which accumulate 
to +oo and — oo. The nonreal spectrum of T is bounded and consists of at most 
finitely many normal eigenvalues which are symmetric with respect to the real line. 
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Before we prove this theorem a preparatory lemma on the resolvent set of T will 
be proved. 

Lemma 3.3. The set p(T) is nonempty. 

Proof. If is not a normal eigenvalue of A, then € p(A) and it follows from 
T = R- 1 A that T- 1 = A~ l R G C(L 2 (n)) holds, i.e., G p{T). Therefore, assume 
that € o-(A), that is, is an isolated eigenvalue of finite multiplicity of A by 
Theorem 13. II The restriction 

B := A \ (domAn (kerA)" 1 ) 

of A on the orthogonal complement of ker A in L 2 (il) is regarded as a nondensely 
defined symmetric operator in L 2 (Q) with finite equal defect numbers. Note that B 
is injective and that rani? = (ker A) 1 - is closed and has finite codimension. Hence 
there exists a selfadjoint operator A in L 2 (f2) which is an extension of B such that 
G p(A). Furthermore, since B is a finite dimensional restriction of both A and A 
it follows that 

dimran ((A - A) -1 - (A - A) -1 ) < dimker A < oo 

holds for all A G p(A) n p(A) and hence cr(A) is semibounded from below and 
consists of normal eigenvalues. 

The operator T := R~ X A is a selfadjoint operator in the Krein space (L 2 (il), [■, ■]) 
and from G p{A) we conclude G p(T). Furthermore, since A is semibounded 
from below and [Tf, g] — (Af, g) holds for all f,g€ dom T = dom A, it follows that 
the form [T-, ■) has finitely many negative squares. It is easy to see that T and T are 
both finite dimensional extensions of the nondensely defined operator S := R~ X B. 
Now p(T) 7^ follows form a slight modification of |T51 Proposition 1.1], see also 
Corollary 2.5]. □ 

Proof of Theorem \3.SX Observe that by Theorem 13.11 the resolvent (A — A) -1 is 
compact for all A G p(A) and that Lemma T3.3I implies p(T) n p(A) ^ 0. A simple 
computation shows that the relation 

(T - A)" 1 = {A- A)- X i? - \(A - A)- 1 ^ -R)(T- A)" 1 

holds for all A G p(A) n p(T), and since the right hand side is a compact operator 
the same holds for the left hand side. Hence <r(T) consists of normal eigenvalues. 
As the negative spectrum of A consists of at most finitely many normal eigenvalues 
the form [T-, ■] = (A-, ■) has finitely many negative squares and it follows from the 
general results in [T21 EH ES] that the nonreal spectrum of T consists of at most 
finitely many normal eigenvalues which are symmetric with respect to the real line. 
Finally, the assumption that the sets fi+ and fi_ in (|2.2p have positive Lebesgue 
measure imply that the indefinite inner product [•, •] in (12. 4[) has infinitely many 
positive and negative squares. The reasoning in [TSl Proof of Proposition 1.8] shows 
that the positive spectrum of T, as well as the negative spectrum of T is of infinite 
multiplicity, and hence the real eigenvalues of T accumulate to +oo and — oo. □ 

4. Spectral properties of indefinite elliptic operators on unbounded 

DOMAINS 

In this section we study the spectral properties of the indefinite elliptic operator 
T in (|2.6p on an unbounded domain Q C M™. Since for an unbounded domain 
the embedding of Hq(H) into L 2 (fl) is in general not compact also the resolvent 
of the selfadjoint operator A in (|2.3p is in general not compact and hence essential 
spectrum may occur. Only the following weaker variant of Theorem 13.11 holds . 
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Theorem 4.1. The spectrum of A is bounded from below and accumulates to +oo. 

If the lower bound min a (A) of the spectrum of A or the lower bound min cr css (A) 
of the essential spectrum of A is positive, then it is known that T = R~ 1 A is 
positive in the Krein space (L 2 (Q), [•,•]) or has a finite number of negative squares, 
respectively. For the convenience of the reader we recall these and some other facts 
in Theorem l4.2l and Theorem l4.3l below. The proofs of the statements are essentially 
contained in [H [EH 131 133 , see also [31 Theorem 3.3], [TQ Theorem 3.1] and Q3 
Proposition 1.6]. 

Theorem 4.2. Ifmino-(A) > 0, then the spectrum of T is real, € p(T), and T 
has positive and negative spectrum, both of infinite multiplicities. 

Theorem 4.3. // min o~ css (A) > 0, then the essential spectrum of T is real and 
T has positive and negative spectrum, both of infinite multiplicities. The nonreal 
spectrum of T is bounded and consists of at most finitely many normal eigenvalues 
which are symmetric with respect to the real line. 

In the next step the assumption min <7 ess (A) > will be dropped. The following 
considerations and Theorem 14.41 below are partly inspired by general results on 
sclfadjoint operators in Krein spaces and the regularity of the critical point 00 from 
[T4l HM 133] and [TUl Proof of Theorem 5.4]. Fix some v < min a(A) and define 
the space T-L s , s £ [0, 2], as the domains of the |-th powers of the positive operator 
A - v, 

H s :=dom((A-v)i), se[0,2]. 

Note that H = Ho, domA = H2, and the form domain of A is Hi- The spaces 
W s become Hilbert spaces when they are equipped with the usual inner products, 
the induced topologies do not depend on the particular choice of v < miner (A); cf. 

ED- 

The following theorem is one of the main results of the present paper. Under an 
additional abstract condition from |16j it will be shown that the nonreal spectrum 
of the indefinite elliptic operator is bounded. Roughly speaking, this condition is 
satisfied in special situations when chosing W — R; cf. Lemma 15.11 in the next 
section. 

Theorem 4.4. Assume that miner css (j4) < and that the following condition holds: 

(I) There exists an isomorphism W in L 2 (0) such that RW is positive in 
L 2 (n) and WH S C H s holds for some s £ (0, 2]. 

Then the nonreal spectrum of T is bounded. 

Proof. 1. In this step of the proof we construct an indefinite elliptic operator T v 
which is a bounded perturbation of the indefinite elliptic operator T and which 
induces (via its spectral decomposition) a new equivalent norm || • ||^ on L 2 (f2). 

For this fix some 77 < min a(A) and consider the elliptic differential operator A v 
defined by 

- d df 

A nf '■= ( A - V)f = - ^2 'Q x ~ ajk 'dx~ k + ( a ~ ^ G domA v = domA. 

j,k—i ^ 

Clearly A v is a positive selfadjoint operator in the Hilbert space L 2 (Vl) and hence 
the indefinite elliptic operator 



r , 1 ( V- " 3/ 



T vf := ~ 7. a jk^ ^ ( a - 7 ?)/ h / € domT v = dom^, 
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is nonnegative in the Krein space (L 2 (fl), [-, •]), the spectrum <r{T v ) is a subset of 
K and G p(T v ); cf. Theorem 14.21 Note that T, ( and T are connected via 

(4.1) T ri = R- 1 A ri = R- 1 A-r ] R- 1 =T-V, V:=r]R~\ 

and that the perturbation term V in (|4.1j) is bounded. 

By [34, 35 T n possesses a spectral function defined for all bounded subintervals 
of the real line. As a consequence of condition (I) and [TH Theorem 2.1 (iii)] (see 
also [13]) it follows that oo is not a singular critical point of the operator T n and 
therefore also the spectral projections E + and corresponding to the intervals 
(0, +oo) and (— oo,0) exist. Moreover, as T n is nonnegative in (L 2 (fl), [•, •]) the 
spectral subspaces (E±L 2 (fl), ±[-, •]) are both Hilbert spaces and L 2 (f2) can be 
decomposed in 

(4.2) L 2 (il) = E + L 2 (n)[+}E_L 2 {fl). 

We point out that the subspaces E±L 2 (£l) differ from L 2 (Q,±) and that within this 
proof the subscripts ± are used in the sense of (|4.2[) . From the properties of the 
spectral function it follows that T v has diagonal form with respect to the space 
decomposition (|4.2I) . 



T v = 



TV 



and that the spectrum of T Vj ± is contained in . The perturbation term V = r]R 
in (|4.ip admits the matrix representation 



V 



V u V 12 
Vty V22 



with respect to the decomposition (|4.2I) . Together with (|4.1I) we then have 
(4.3) T = T v + V = T n 



Vu V 12 
V21 V22 

In the following we write functions x, y £ L 2 (fl) in the form x = x + + .t_ and 
y = y + + y_, where x±,y± € E±L 2 {Q)- 1 cf. (|4.2|) . We emphasize that x± are the 
components of x with respect to the space decomposition (|4.2p and that x± do 
not coincide with the restrictions of the function x onto il±. Since the spectral 
subspaces (E±L 2 (fl), ±[-, •]) are Hilbert spaces the inner product (■, -)^ defined by 

(x,y)„ := [x+,y+] - [x_,j/_], x,y S i 2 (fi), 

is positive definite. Furthermore, this scalar product is connected with the usual 
scalar product (•, •) on L 2 (il) via 

(x,y)„ = [E+x,E+y] - [E_x,E_y] = [(E+ - E_)x,y] = (R(E+ - EJ)x,y). 

Therefore, as R(E + — _E_) is an isomorphism, the norms || • || and || • ||^ induced 
by the scalar products (•, •) and (v)~) respectively, are equivalent. In particular, 
with v := \\R(E + — _E_)|| _1 we have 

(4.4) ||.t|| < v^lkll- for all x e L 2 {Vt). 

2. In this step it will be shown that for sufficiently large \/j,\, fi G C\K with Re /.i < 
the operator + + Vu — ji is invertible and that the estimates 

(4.5) \\(T lh+ + V 11 -^)- 1 \U<\ and ||(T„, + + V u - m) _1 ^2|U < \ 

hold. By replacing Vu and V12 in the reasoning below with V22 and V21, respectively, 
it follows that for sufficiently large \fi\, ji € C\K with Re/i > the operator 
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T, ; _ + V22 — is invertible and that the estimates 

IKT^. + Vaa-M) -1 !!- < ~ and ||(r,,_ + V 22 - /i) -1 ^!^ < 5 
are valid. 

In the following we assume that the entry V12 in the perturbation term V is 

nonzero (otherwise the first estimate in (|4.5[) follows with S > 2 and r = 5+ 1 1 V11 1| 

in the argument below; the second estimate is trivial). Choose S > such that 

'2+llViilL „ , II Villi 



(4.6) 5+ || Vnll ~ > max 



IV2IU ' IIV2I 



and define the constant r by 

(4.7) r:= (5 + ||Vii||~)||Vi 2 ||~ 

Let e C\M with Re^i < and |/z| > r. Since cr(T,^ + ) C R + it is clear that 

dist (fi,a(T v+ )) > t 
holds and therefore we have ||(7?7,+ — A*) _1 ||~ < t^ 1 . This implies 

||Vii(T J ,, + - A1 )- 1 |U < -\\Vu\\~ 

T 

and it follows from (g^D and g^l that ||V U (T,, + - Ai) _1 ||~ < 1- Therefore the 
operator / + Vn{T, h+ — /i) _1 is boundedly invertible and the norm of the inverse 
can be estimated by 

||(7 + V n (T rh+ - Mr 1 )" 1 !!- < (l-~|| Vnll 

It follows that also the operator 

T v ,+ + Vn -n=(l+ Vii(T„, + - M) _1 )m,,+ - m) 
is boundedly invertible and we conclude 

1 A ., \ 1 



(4.8) ikt^ + Vu-m) -1 !!- < - 1--IIV1 



T V T J T — IV 



Since by (|4.6|l and (|4.7p r - ||Vu||~ > 2 we obtain the first estimate in (|4.5p . 
Furthermore, as a consequence of (|4.6p and (|4.7p we have 

IIV12IU _ HV2IU 1 



r-iiviiu (5 + ||Vn|U)||Vi 2 |U- nviiu 2 

and therefore (|4.8p yields the second estimate in (|4.5p . 

||(T„ 1+ + Vn - ^"VialU < < ~ 

t - || Vn\\~ 2 

3. Next we verify the inequality 
(4.9) ||(T-aOs||~> 




|Im fi\ 




for x £ domT and all sufficiently large \fi\, /j, € C\E. Observe first that for 
x G dom T we have 

[(T — fi)x, x] — [(T — Re fx)x, x] — ilmij[x, x] 
which together with (|4.4I) implies 

\Im^\\[x,x]\ < \[(T-»)x,x]\ < \\(T - n)x\\\\x\\ < v\\(T - rix\\„\\x\\„ 
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and hence 

(4.10) jj^ll(T-/i)z||~H~ > ±[x,x]. 

On the other hand, when we consider the equation (T—fj,)x = y with x = x + +x- , 
V = y+ +V-, x±,y± £ E ± L 2 (n), that is, 

(T v>+ + Vu - n)x+ + V12X- = y + 
V 2 ix + + (T Vi - + V22 - A«) x - = V- 

(see (|4.3[) ). then we conclude with the help of the estimates from step 2 that for 
sufficiently large \i € C\R with Re^ < 

||as+||~ < \\(T n+ + V u - rf^V+W- + \\(T n+ + V u - n^V^x-W^ 

< \\\v+\\~ + ^ Mas— II— < + 

holds and that for sufficiently large € C\R with Re/x > 

IMU < \\(T v _+V 2 2-ri- 1 y-\\~ + \\(T n _+V22-li)- l V2 1 x + \\„ 

< 2lltf-ll~ + 5ll a: +ll~ ^ 2 1 b-ll~ + t^IMU 

holds. Since ||?/±||~ < ||y||~ = ||(T — we have 

(4.H) \\x±\\i < \\\{T rfxWl + \\\x\\l + \\\{T fix\U\x\\„ 

for sufficiently large |/i|, [i £ C\R with Rc/i < and Re/i > 0, respectively. From 
||x+||l + ||a;_||l = \\x\\l we obtain 

±[a:,a;] = ±||s + ||l T = bill - 2||a: T ||* 

and together with (|4.11[) we conclude 

±{x,x) > \\\x\\i - ±\\(T - n)x\\l - \\(T - n)x\Ux\\^ 

for sufficiently large fi £ C\R with Re/z > and Re fi < 0, respectively. 
Together with (|4~TU| this leads to 

j^jlKT - M>||~NI~ > ±\\x\\l \\\(T-n)x\\l || (T - /x)*IWMU 

for all sufficiently large |/x|, /x 6 C\R. In other words, ||(T — fi)x\\^ satisfies the 
quadratic inequality 



\\(T-»)x\\l + 2^1+^—) \\x\\„\\{T-n)x\\„-\\x\\l >0. 

Hence it follows that (|4.9I) holds for all x £ domT and all /i £ C\R with |/x| 
sufficiently large. 



4. Let A € C\R such that (|4.9[) is satisfied with fi = A and fj, = A. Then we have 
ker(T — A) = {0} and ran (T — A) is closed as T is closed. Furthermore, since T is 
selfadjoint in the Krein space (L 2 (fl), [•, •]) it is clear that ran (T— A)W = ker(T — A) 
holds. As || (T — A)x||^, x £ domT, satisfies the same estimate as ||(T — A)x||^ in 
(|4.9[) this implies that also ker(T — A) is trivial. Therefore T — A is bijective, i.e., 
A £ p(T). Since this is true for every A = fi which satisfies f|4.9[) we conclude that 
the nonreal spectrum of T is bounded. □ 
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5. Spectral properties of indefinite elliptic operators on R n 

In this section we consider the case SI = M™ and we assume that the subsets 
il± = {x £ 1™ : ±r(x) > 0} consist of finitely many connected components with 
compact smooth boundaries. In particular, this implies that one of the sets f2± is 
bounded and one is unbounded, and that the boundaries <9f2+ and <9S1_ coincide. 
Here and in the following we discuss the case where Sl_ is unbounded and f2+ is 
bounded and we denote the boundary dfl± by C. The simple modifications of the 
results below to the other case are left to the reader. Since the weight function 
satisfies r, r _1 £ L°°(R n ) the restrictions r±,r± 1 belong to L°°(S7±) and hence the 
multiplication operators R±f± = r±f± are isomorphisms in L 2 {Q,±) with inverses 

R± 1 f± = r ± 1 f±, f±eL 2 (n ± ). 

Let us now assume that the coefficients ajk £ C°°(M. n ) in (|2.I[) and their deriva- 
tives arc uniformly continuous and bounded, and that (as before) a £ L°°(W l ) 
is real valued. An essential ingredient for the following considerations is that by 
elliptic regularity and interpolation 

domi = domT = H 2 {R n ) and H s = H s (R n ), s £ [0, 2], 

holds; cf. [2 El [Mllll], [33 Condition 3.1] and ([231), ([22]). Here H s (R n ) is the 
Sobolev space or order s. The spaces consisting of restrictions of functions from 
H s (R n ) onto f2± are denoted by H s (fl±). In the next lemma and remark we give 
simple sufficient conditions for the weight function r such that condition (I) in 
Theorem gjH holds. 

Lemma 5.1. Assume that for some s £ (0, |) the spaces H s (n + ) and H s (fl^) are 
invariant subspaces of the multiplication operators R + and i?_ , respectively. Then 
H s {R n ) is an invariant subspace of the multiplication operator R, and condition (I) 
in Theorem \4--4\ is satisfied with W replaced by R. 

Proof. Let s be as in the assumptions of the lemma and let / £ H s (R n ). Then the 
restrictions f± of / onto f2± are functions in H S (Q,±) and therefore, by assumption, 
the functions g± := r±f± also belong to H s (il±). As < s < the continua- 
tions g± of g± by zero onto R" both are in H s (R n ); cf. [2HJ Theorem 1.4.4.4 and 
Corollary 1.4.4.5] and note that the proofs of these statements in [29 also cover 
the case of an unbounded domain with a compact smooth boundary. Therefore 
Rf = rf = g+ +g~ £ H S (W 1 ) and hence £P(R") is invariant for R. Furthermore, 
R is an isomorphism in L 2 (R") and the estimate R 2 > essinfr 2 > holds, i.e., R 
possesses all the properties of the operator W in condition (I) in Theorem 14.41 □ 

Remark 5.2. If, e.g., the function r is equal to a (negative) constant outside some 
bounded subset of R n and belongs to the Holder space C 0,a (R n ) for some a > 0, 
then it follows from [291 Theorem 1.4.1.1] and a similar argument as in the proof 
of Lemma 15.11 that H s (il±), s £ (0,a), are invariant subspaces of R±. 

For completeness we state the following immediate consequence of Theorem 14.41 
and Lemma 15.11 

Corollary 5.3. Assume that mincr ess ( J 4) < and R±(H S (f2±)) C H s (fl±) holds 
for some s £ (0, |). Then the nonreal spectrum of T is bounded. 

In the following theorem we obtain more precise statements on the qualitative 
spectral properties of T. The proof is based on a multidimensional variant of Glaz- 
manns decomposition method from the theory of ordinary differential operators, 

see, e.g., [a nana us Gig. 

Theorem 5.4. Assume that min<7 ess (A) < 0. If p(T) =/= 0, then the essential 
spectrum ofT is real, bounded from above, and o~ css (T)n [0, oo) ^ holds. Moreover, 
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the nonreal spectrum of T consists of normal eigenvalues which are symmetric with 
respect to the real line and which may accumulate to points in o~ css (T). 

If, in particular, R±(H S (Q±)) C H s (n±) holds for some s G (0, then the 
assumption p(T) ^ is satisfied, the above assertions hold and the nonreal spectrum 
of T is bounded. 

Proof. Besides the operators A and T we will make use of the selfadjoint elliptic 
differential operators 

(5.1) A±f± := £(/±), dom A ± = H 2 (fl ± ) n H^(Sl ± ), 
in L 2 (VL±) and the weighted differential operators 

(5.2) B ± f ± := £±(/±) = —£(f±), domB± = H 2 (Q ± ) n H%(Sl ± ), 

r± 

which are selfadjoint in the weighted L 2 -space L 2 (£l±,±r±), where the (positive 
definite) scalar products (•, -)± arc defined by 

(5-3) (f±,9±}±:= [ f±(x)ljljx)(±r ± (x))dx, f±, g± G L 2 (fl ± ). 

Jn± 

Observe that the orthogonal sums A + © A- and B + © £?_ are selfadjoint operators 
in L 2 (£l) and L 2 (fl,r), respectively. Furthermore, since the boundary C is compact 
and smooth it can be shown that the resolvent differences 

(5.4) (A ~ X)- 1 - ((A+ (B A_) - Xy 1 , X G p(A) n p(A + © AJ), 
and 

(5.5) (T - A) -1 - ((-B+ B-) — A) 1 , Xe p(T)np{B+®B_), 

are compact operators in L 2 (W l ); cf. [12] and Theorem 15 . 71 below. 

Recall that the spectra of A± are bounded from below and, moreover, as 
is assumed to be bounded, the spectrum of A + consists of normal eigenvalues; cf. 
Theorems 13.11 and 14.11 Furthermore, the differential operators in (|5.1[) and (|5.2|) 
are connected via 

(5.6) B± = R± 1 A±, 

where R± are the multiplication operators with the functions r± . One verifies that 
for A G p(A+) n p{B + ) the resolvents of A + and B + are connected via 

(B+ - A)- 1 = (A+ - X)- 1 R + - X(A + - A) _1 (7 - R+)(B + - A)" 1 

and since (A + — A) -1 is compact the same holds for the resolvent of B + ; cf. the 
proof of Theorem 13.21 Thus the spectrum of B + is also bounded from below and 
consists of normal eigenvalues which accumulate to +oo. 

Next the spectrum of £?_ will be described in terms of the spectrum of A- . Since 
the resolvent difference in (|5.4p is compact and (T 0SS (A + ) = wc conclude 

a css (A) = a css {A + © AJ) = cr css (^-)- 

Furthermore, the assumption mincr eS s(^4) implies that A_ is bounded from 
below with negative lower bound 

(5.7) v := min cr(A_) < min (T ess (j4_) < 0. 

Denote the usual scalar product in L 2 (f2_) by (-, •)_ and let 7 be the supremum of 
the weight function r_ on Then we have 7 < and (— r_(x)) _1 < (— for 
all x G f2_. Moreover, from the estimate 

(/-,/-)-=/ -^If-ix^i-r^dx <—(/_,/_)_, /€L 2 (n 
Jn_ -7 
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we obtain together with (|5.3[) and (|5.6p that 



/-,/->- = /_,/_)_ < -!/(/_,/_)_ < -</-,/->- 

7 

holds for all /_ € dom_B_, i.e., the spectrum cr(B-) and the essential spec- 
trum o~ css (B-) are bounded from above by the positive constant £ Observe 
that max<7 eS s(-B-) > holds, since otherwise min er oss (— _B_) is positive and A- = 
(— implies that also min cr ess (A_ ) is positive which contradicts (15.71) . 

Summing up we have shown that the essential spectrum of B + ffi £?_ is real, 
bounded from above and a ess (B + ffi J3_) n [0, oo) 7^ 0. Since the resolvent difference 
(|5.5[) is compact we obtain er css (T) = a css (B + ffi £?_) = er css (£>_) which together 
with Corollary 15.31 yields the statements. □ 

In the following we will show that the nonreal eigenvalues of T and the corre- 
sponding eigenspaces can be characterized with the help of Dirichlet-to-Neumann 
maps associated to the restrictions of the elliptic differential expression C on Q±. 
For this, recall first that the mapping C°°(Q±) 3 f± h-> {.f±\c, §^lc} extends to a 
continuous surjective mapping 

(5.8) H 2 (il ± ) 3 f ± ^ |/ ± |c, \ G H 3/2 (C) x H^iC), 

where ^\ c -= Z)£*=i n ±.il^lc and n ±( x ) = ( n ±A x )> ■ ■ ■ ,n±,n{x)) is the unit 
vector at the point x G C pointing out of f2±. The next simple lemma is based on 
a standard decomposition argument. For the convenience of the reader we provide 
a complete proof. 

Lemma 5.5. For A G C\R and y> G H 3 / 2 (C) there exist unique functions f±,\(<p) G 
if 2 (r2±) swc/i i/iai 

C±f±,\( ( P) = A/±,a(v) /±,a(v)|c = <£>< 

Proof. It is sufficient to show that for A G C\K the linear subspace 

(5.9) S := {h+ ffi fc_ G # 2 (^+) tf 2 (ft_) : fe+| c = h_| c } 
admits the direct sum decomposition 

(5.10) S = {g+®g-eS: g±\ c = 0} + {/i+,A © h-,A € 5 : £±/i±, A = A/i±,a}. 

In fact, it follows from (|5.8|) that the trace map h >-> h\c defined on S in (15.91) maps 
onto H 3 / 2 (C) and since the first term on the right hand side of (I5.10[) is its kernel 
it follows that the trace map maps the second term on the right hand side of (|5.10l) 
bijectively onto H 3 / 2 (C). 

In order to prove the decomposition (|5.10[) note first that the inclusion D in 
(I5.10[) holds. Hence it remains to verify the inclusion C. For this let h+ © h- 6 5 
and A G C\M be fixed. Since the boundary C of 0± is assumed to be compact and 
smooth it follows that the differential operators B± in (15.21) are defined on 

(5.11) domS± = H 2 (n ± ) n flg(n±) = {/ e H 2 (n ± ) :f±\ e = o}. 

Hence the first set on the right hand side of (|5.10l) coincides with dom {B + ffi £?_ ) . 
Since the spectrum of B + © i?_ is contained in R (see the proof of Theorem I5.4[) 
B + ffi £>_ — A, A G C\K, is a bijection from its domain onto L 2 (M. n ). Thus there 
exists g+ ffi g- G dom (B + ffi B_) such that 

(£+ - X)h + ffi (£_ - \)h- = (B + - X)g+ ffi (B- - X)g- 

Therefore C±(h± — g±) = \(h± — g±) and hence 

h+ ffi h- = g+ © g- + ((h+ - g+) ffi (h- - g-)) 
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shows that the inclusion C in (15.101) is also valid. The sum in (|5.10p is direct 
since a(B + © £?_) C R; indeed, each element in the intersection of the sets on the 
right hand side of (|5 . 10[) would be an eigenfunction of B + © i?_ corresponding to 
A G C\R. □ 

For A G C\R, tp G H 3 / 2 (C) and f±,\(<p) G H 2 (n ± ) as in Lemma [531 we define 



(5.12 



) M(X):H^(C)^H^(C), <p ^ ^I±AEl 



c dv. 



Roughly speaking M is the sum of the Dirichlet-to-Neumann maps associated to 
C± which map the Dirichlet boundary values of solutions of C±f± = A/± onto 
their Neumann boundary values. A similar function in a "definite" setting appears 
also in |43) . In the next theorem we show how the nonreal eigenvalues of T can be 
described with the help of the function M. 

Theorem 5.6. Let the operator function A <— > M(A) be defined as in (|5.12j) and 
assume that p(T) 7^ 0. Then 

a(T) n (C\K) = {A G C\R : kerM(A) ^ {0}} 

and ker(T - A) = {/ G H 2 {K n ) : M(A)/| C = 0} for all A G C\R. 

Proof. Assume first that A G C\R belongs to the spectrum of T. Then, by Theo- 
rems 14.31 and 15.41 the point A is a normal eigenvalue of T and hence £/ = A/ holds 
for some nontrivial / G domT = H 2 (W l ). In particular, the restrictions f± of / 
onto f2± belong to H 2 (£l±) and we have 

df- 



(5.13) C±f±=\f±, f+\ c = f-\c, and ^± 



d 



By (|5.8p we have <p := f±\c G H 3 / 2 (C) and hence f± = f±,\{<p) in the notation of 
Lemma 15.51 The third property in (|5 . 1 3[) implies 

ov + OP— ov+ OV- 

and hence tp G kerM(A). Furthermore, ip is nonzero, as otherwise f± G H 2 (il±) 
would be nontrivial solutions of the Dirichlet problems C±f± = A/±, f±\c = 0, 
which do not exist due to A g" R. In other words, since the selfadjoint operators 
B± in (|5.2p do not have nonreal eigenvalues we conclude ip 7^ 0. 

For the converse let A G C\R and ip G kerM(A) with ip 7^ 0. By Lemma [5.51 
there exist unique functions f± t \(<p) € H 2 (Q±) such that £±f± > \(<p) = Xf± t \(ip) 
and f±.x(<p)\c = V hold. Since M(X)tp = we have 

(5.14) df^ = _df^p)_ 

ov + ov- 

Define the function / = /+ © /_ G L 2 (R") by /± := /±,a(p) and let 3 G domT. 
Then / ^ and 

[Cf,9]-[f,Tg] = (lf,g)-(f,£g) 
' ; = (£+/+, 5+ )+ - (/+,£+<?+)+ + (t-f-,g-)- - (f-,£-g-)-, 

where [•, •] is the indefinite inner product in (|2.4[) . (•, •) is the usual scalar product 
in L 2 (R n ) and (v)± denote the scalar products in L 2 (fl±). Since the function 
g G domT satisfies 
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it follows from Green's identity, /+|c = /_ |e> and (|5 . 14[) that (|5 . 1 5[) is equal to 

,g-\e] =0. 



f+\c— J - I — c >9+\c I + I f-k'-faT 



dv 



+ 



This is true for any g £ domT and since T is selfadjoint with respect to [•,•] 
we conclude from [Cf, g] = [f,Tg] that / £ domT and Tf = Cf. Moreover, 
from C±f± = A/± we obtain / £ ker(T — A), i.e., A is an eigenvalue of T with 
corresponding eigenfunction /. □ 

The next theorem provides a variant of Krein's formula which shows how the 
resolvent of the indefinite elliptic operator T differs from the resolvent of the or- 
thogonal sum of the weighted differential operators 

B±f±=£±(f±) = —£±(J±), domB ± =H 2 (n ± )nH^(n ± )- 
r± 

cf. (15. 2p and (15 . . The operators T and B + © B_ are viewed as operators in 
L 2 (R"). We note first that the statements in Lemma T5. 5 1 and Theorem 15.61 remain 
true if the set C\K is replaced by the resolvent set of the operator B + © £?_ . This 
set contains C\R and may also contain subsets of the real line. For A £ p(B + (&B-) 
define the mapping 7(A) : L 2 (C) -> L 2 (R") by 

7(A)? := f+,x(<p) 8 f-,x{fp), dom 7 (A) = H 3 / 2 (C) 7 

where f±.\((p) are the unique solutions of C±u± — Xu±, u±\c = <f>\ cf. Lemma I5~5l 
Theorem 15.71 is an indefinite variant of [4, Theorem 4.4 (ii)] and can be proved 
in almost the same way. Therefore we only indicate some ideas of the proof and 
refer the reader to [U §4] for the details, see also [5]. Recall that the multiplication 
operator R is an isomorphism in L 2 (M. n ). 

Theorem 5.7. For all A £ p{T) n p(B + © B_) the difference of the resolvents of 
T and B + © £?_ is a compact operator in L 2 (IR ra ) given by 

(5.16) (T - A)" 1 - ({B+ © B-) - Xy 1 = 1 {\)M{\)- l 1 {\)*R. 

Proof. A slight modification of 4, Proposition 4.3] yields that 7(A) is a densely 
defined bounded operator from L 2 (C) into L 2 (W l ) and that the adjoint operator 
7(A)* : i 2 (R") -> L 2 (C) has the property 



7(A)*i?((B + ffiB-)-A)/ = -§^ 



c dv- 



where / = /+©/_ £ dom(B + © B-). In particular, ran7(A)* C H l / 2 (C) and 
together with Lemma lipTSl Theorem 15.61 and (|5.12[) we conclude that the right hand 
side in (|5.16[) is well defined for all A £ p{T) n p(B + © B_). The same reasoning 
as in the proof of [U Theorem 4.4 (ii)] shows the relation (I5.16P for the difference 
of the resolvents of T and B + © £>_ in L 2 (M. n ). Moreover, it follows from [55] in 
the same way as in [H Corollaries 3.6 and 4.6] that for A £ p(T) D p{B + © B_) the 
closure of M(A) _1 in L 2 {C) is a compact operator in L 2 (C). Therefore, the right 
hand side of (|5.16p is a compact operator in L 2 (M. n ). □ 

Remark 5.8. We note that the resolvent difference in (|5.16[) is not only compact 
but belongs to certain Schatten-von Neumann ideals that depend on the dimension 
n; cf. [H EH [301 [37]- 
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